Skip to content

[FEATURE] Render every page to Markdown alongside HTML - #1410

Open
linawolf wants to merge 16 commits into
mainfrom
task/markdown-output-format
Open

[FEATURE] Render every page to Markdown alongside HTML#1410
linawolf wants to merge 16 commits into
mainfrom
task/markdown-output-format

Conversation

@linawolf

@linawolf linawolf commented Sep 12, 2026

Copy link
Copy Markdown
Member

Part of #1400.

Tools that read our documentation programmatically want the content without the
surrounding HTML. The reStructuredText source is not good enough: includes,
substitutions and interlinks are unresolved there, and directives only become
content during rendering.

A new package typo3-docs-theme-md registers an md output format on the same
AST and pipeline as HTML, so RenderContext writes Feature.md beside
Feature.html. The theme ships it; render-markdown="false" switches it off.

Markdown is GitHub Flavored — admonitions and version changes become alerts,
tables become pipe tables. Links are permalinks built from
interlink_shortcode, because a downloaded file has to keep working.

Verified against 39 real manuals: the core changelog, the TCA, TypoScript,
ViewHelper and exception references, eleven guides and tutorials including
TYPO3 Explained, the 21 system extensions that ship their own documentation,
and the news and content-blocks extensions as third-party cases whose
interlink shortcode is a composer name. 10264 Markdown files.

  • No unrendered directive anywhere. A node without a template leaves an HTML
    comment naming it; the rendertest started at 243 of those, and every manual
    now reports none.
  • Every file opens with front matter that parses as YAML, and every manual has
    exactly one page marked as its start.
  • Every permalink into a manual's own inventory resolves, and carries the
    version wherever the manual declares one: all 92 in the TCA reference and 382
    of the 383 across the system extensions, none of the 3912 in the changelog,
    which is published once.
  • No HTML entities beyond the ones the manuals document on purpose — pages
    about htmlentities(), htmlparser or XML examples, where HTML shows the
    same characters.
  • Nothing the HTML carries that the Markdown lacks. Comparing the text of both
    page by page, what is left is the HTML splitting identifiers across
    syntax-highlighting spans, and controls: "Copied!", "Edit on GitHub", the
    breadcrumb.

Rendering Markdown alongside HTML costs about 2%, which is less than the spread
between two identical runs — 126s to 142s for the same changelog — so the real
figure sits inside the noise.

Fifteen commits. The first three are the render target, the setting with its
download button and the rel="alternate" link, and the escaping together with
confval, rubric and the text roles. The rest add one directive family each, in
the order the manuals turned them up, and close with the front matter that
tells a file which manual, version and source it came from. Four are bugfixes:
an image lost its alt text, the file text role lost its code formatting, a code
block lost the caption that says which file the snippet belongs in, and the
permalinks named no version, so every link in a 13.4 manual led to whatever is
current later.

This supersedes #1401, which added the same rel="alternate" link pointing at
the copied reStructuredText source.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) noreply@anthropic.com
Claude-Session: https://claude.ai/code/session_019EuSAzF5qw81NzPusWy2hC

Tools that read our documentation programmatically want the content
without the surrounding HTML. The reStructuredText source is not good
enough: includes, substitutions and interlinks are unresolved there, and
directives only become content during rendering.

This is a render target on the same footing as HTML - same AST, same
pipeline, its own templates. A new package typo3-docs-theme-md registers
an "md" output format with 40 templates and its own node renderers, so
RenderContext writes Feature.md beside Feature.html. The theme ships it,
so no second extension has to be configured.

Markdown is GitHub Flavored: admonitions and version changes become
alerts, tables become pipe tables. CommonMark has no tables at all.

It renders always. That cannot be done by prepending "output_format":
an explicit value replaces the default ["html", "interlink"] rather than
extending it, so a project configuring nothing would get Markdown and no
HTML. Appending in the compiler pass leaves every other format
untouched - a project rendering only "singlepage" or only "rst" keeps
doing that. Naming "md" explicitly still wins, so a Markdown-only render
stays possible.

Links are permalinks built from interlink_shortcode, because a
downloaded file has to keep working. A page linked without an anchor is
in this same manual, so its document entry supplies one; "#" is the page
itself. Anchors are normalised the way getPermalink() does it: a URL
fragment keeps the casing of the element id, while the inventory
registers the target reduced. Where no permalink can be built the link
stays relative and that is logged.

A toctree renders as the list of links it is, unless it is :hidden: -
the same rule the HTML output applies. A node with no Markdown template
leaves an HTML comment naming it, so the remaining gaps are greppable
rather than silent.

The HTML head points at the Markdown with rel="alternate", which
supersedes #1401.

Verified against the TYPO3 core changelog, 3912 entries: four confval
directives left unrendered, no HTML entities, and all 12622 permalinks
resolve to a target in the generated inventory. Rendering Markdown
alongside HTML costs 8.7% (127s to 138s).

Part of: #1400

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019EuSAzF5qw81NzPusWy2hC
render-markdown="false" drops the Markdown files, the head link and the
menu entries together.

Adds a "Markdown" download button beside "Give feedback" and "Edit on
GitHub", and "View as Markdown" in the options menu.

The download is named from the manual and the page instead of the URL,
where every overview page arrives as "index.md". The name comes from the
document's label, not its title: titles repeat, so an entry backported
to three versions shares one title anchor. Using it collided 29 of 3912
names and pointed a :doc: link at the wrong version.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019EuSAzF5qw81NzPusWy2hC
Source prose is not Markdown. Unescaped, "__dunder__" turned bold,
"array[0](x)" turned into a link and a leading "#" into a heading. Only
what can start a construct is escaped: an underscore inside a word is
not emphasis in CommonMark, so escaping every one added 2287 backslashes
to the core changelog for nothing.

confval renders as a list item: indenting under a bare paragraph is an
indented code block, not containment, so a definition had no visible end
and nested confvals did not nest. The name is bold and the property
labels italic; neither wraps a code span, which not every renderer
parses.

rubric becomes bold via a GeneralDirectiveNodeRenderer that looks up a
template per directive name, the way HTML does. Text roles dispatch by
name too, so ":code:" and the default role keep their markup.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019EuSAzF5qw81NzPusWy2hC
composer normalize sorts "T3Docs\Typo3DocsThemeMd\" before
"T3Docs\Typo3DocsTheme\", and the Quality job checks it.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019EuSAzF5qw81NzPusWy2hC
…down

Tabs, cards and accordions are boxes: panels behind a click, a bordered
card, a collapsed stack. Markdown has none of that, so each title becomes
bold and the contents follow one below the other.

A card writes out its image, header, title, body and footer in that order.
A footer given a block can hold a whole list, so it is rendered as blocks
rather than collapsed onto one line.

Three nodes were exempted from the fallback marker as structural, on the
assumption that they pass their content through. They do not: rst-class
dropped its whole content, and field and definition lists lost their
terms. Each one now has a template.

An image read its alt text and its path from properties that do not
exist, so every image rendered as "![]()" pointing at an unnormalised
path. It now reads the same options the HTML template reads.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A console command is a named definition with fields, like a confval, and
is rendered like one: the signature bold as a list item, so the
definition has an end and the arguments and options nest inside the
command they belong to. Usage, arguments, options and help are italic
labels, so the command outweighs the words introducing its parts. A
command whose help text is empty gets no "Help" label.

The overview becomes a pipe table. A command is linked only when it can
be: the anchor of a noindex command is registered nowhere, so a permalink
to it would lead to a 404 rather than to the command.

An argument shows only its description, as in HTML. ArgumentNode also
carries isRequired and a default, but CommandNodeService fills the first
from "is_value_required", an option key, so it is always false, and gives
the second the description. Rendering either would state something untrue
about the command.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A ViewHelper writes out the parts its "display" option asks for, in that
order, as in HTML: doc tags, prose, the link to the source and the
arguments. A deprecated or internal tag becomes an alert, and so does the
note about arbitrary arguments.

An argument is shaped like a confval, which is what the HTML template
builds it as too: the name bold, then type, default and required, then
the description. "Required" reads as true rather than as the "1" that
HTML prints for the boolean.

"Arguments" is bold rather than a heading. The HTML template makes it an
h2, but a section can hold several ViewHelpers, and a real heading would
leave the outline claiming the arguments of the first one end where the
second one starts.

The Markdown parser emits a NullNode where there is nothing to render.
Marking it claimed content was dropped where none exists, so it joins the
nodes the fallback passes through untouched.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
These five directives are layout around content Markdown already has, or
source text HTML turns into a picture.

An hlist is a bullet list in columns and a glossary is its entries behind
an A-Z bar; neither the columns nor the bar survive, the list and the
terms do. A video cannot be embedded and the node carries nothing but the
id, so it becomes a link to itself -- the watch URL, since the privacy
gain of youtube-nocookie.com is about loading the player on our page,
which a link does not do. A diagram and a formula are shown as their own
source, fenced and tagged plantuml and math, which the renderers that
know them pick up.

YoutubeNode, UmlNode and MathNode are not GeneralDirectiveNodes, so the
lookup by directive name does not reach them. Math is registered in the
Markdown package and the other two in the theme, each where the package
holding the node is already a dependency.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A directory tree is a nested list that HTML gives folder icons and
collapse toggles. Markdown keeps the nesting and drops the rest.

The items are not ordinary list nodes -- an item holds its label in its
children and its sub-trees in "subLists" -- so the structure is walked the
same way the HTML template walks it. Passing the node through ran every
name in the tree together on one line.

Each item emits exactly one newline. A blank line between them would make
the list loose, and a renderer then wraps every name in its own paragraph.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
":file:`/etc/passwd`" rendered as bare text in Markdown while every other
code-like role kept its backticks, so a directory tree showed its
directories as code and its files as prose.

A node renderer registered from a template matches with is_a(), so the
first one that fits decides. FileInlineNode reaches InlineCompoundNode
through AbstractLinkInlineNode, and the Markdown package maps that class
generically. prependExtensionConfig() puts the last caller first, and the
Markdown extension was prepended last, so its generic map came before the
theme's own registration and swallowed the role -- leaving the node's
children to render as plain text.

The Markdown extension is now prepended first, which puts the theme's
nodes ahead of the generic map. CodeInlineNode extends InlineNode
directly and was never affected.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Classes, interfaces, traits, exceptions, enums and their members had no
Markdown at all: the markers counted twenty, but the real loss was the
members below them, which ran together into one line of prose --
"A class to do some returning.OtherLibrary\ReturnedClass..."

Each definition is shaped like a confval, the way every named definition
in this theme is: the signature bold as a list item, so the definition has
an end and its members nest inside it. Short properties come before the
description, blocks after it -- a fully qualified name is one line and
leads, a method's "Returns" is prose and follows.

A namespace renders as nothing, as in HTML, where it is an anchor and
nothing else; the namespace is spelled out in every fully qualified name
inside it anyway.

The theme now requires t3docs/guides-php-domain, which it has been
rendering all along through the dependency of another package.

Four escaping bugs surfaced while building this. renderNode() is marked
safe in Twig but a filter after it is not, so a link URL, a definition
list term and a command table cell all reached the output as HTML
entities. The rendertest held eighteen and now holds none.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A caption says which file a snippet belongs in, or whether it is the
before or the after of a migration. Markdown dropped it, leaving
consecutive blocks in the same language indistinguishable: the
StringUtility::multibyteStringPad deprecation showed two bare php blocks
where HTML labels them "Before (deprecated)" and "After (recommended)".

It is now a bold line above the fence, where HTML puts it too.

Found by comparing the rendered TYPO3 core changelog against its HTML:
620 code blocks in 273 of its 3912 files carry one, most of them a file
path such as "EXT:my_extension/ext_localconf.php".

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…down

Rendering the TYPO3 references, guides and tutorials turned up five
directives the rendertest does not contain.

Two carry content and were losing it. The manuals write a configuration
value as "option", not as "confval", and a file definition as
"typo3:file"; both ran together into a line of prose -- "stringEXT:my_
sitepackage/...The path to the Fluid template file". Both are now shaped
like a confval, which is what the HTML templates build them as.

OptionNode is not a GeneralDirectiveNode, so the lookup by directive name
does not reach it; it is registered by class, as in HTML.

The other three are controls, not content, and render as nothing or as
their plain content: the "Edit on GitHub" button beside an included
section, the main menu JSON that only the JSON output writes, and the
list of languages the highlighter knows.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A permalink without a version resolves to the latest stable release. Every
link in the Markdown of a 13.4 manual therefore led into whatever manual
is current when the file is read -- the opposite of what the permalinks
are here for, which is that a downloaded file keeps saying what it said.

Manuals that exist only once take no version; DefaultInventories already
knows which those are, and what it does not know is a third-party manual,
which is versioned. A checkout names itself "main (development)", so the
version is cut back to the bare "main" the URL wants.

A version that is set but cannot go into a URL is reported once per
document. A version that is absent is not: the theme treats it as optional
everywhere else, and there is nothing the author could do about it here.

Other interlinks are unaffected. They resolve through the inventory into a
full URL that already names a version, taken from typo3-core-preferred.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
A Markdown file says nothing about itself: which manual and version it
came from, which source it was rendered from, when, and where its HTML
lives. A tool that has the file has no way back to any of that.

It now opens with YAML front matter carrying title, manual, version,
permalink, source path and the render date, plus "start: true" on the
manual's entry point and "noindex: true" for a page kept out of the
search. Front matter is not CommonMark, but it is what every tool that
reads Markdown expects, and this output exists to be read by tools.

The permalink is the one built for every other link, version suffix and
all, so it names this page independently of where the file ends up and
leads back to the HTML it was rendered from. The source path comes from
where the "Edit on GitHub" link takes it, so an included file names
itself rather than the document that included it. The start page is
recognised by comparing against the project's root entry, because
"Index.rst" is a name several documents in a manual share.

A key whose value is unknown is left out rather than written empty: a
permalink with nothing behind it is worse than no permalink. Values are
always quoted, or a title like "Feature: 12345" would parse as a mapping
and a version like "13.4" as a number.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
@linawolf
linawolf force-pushed the task/markdown-output-format branch from 0d06472 to b256395 Compare September 13, 2026 09:33
@linawolf
linawolf marked this pull request as ready for review September 13, 2026 10:06
The package was not listed anywhere the other five are, so it would never
have been split into its own repository, and the theme required it as
"@dev" while every other dependency inside this repository uses
"self.version" -- a released typo3-docs-theme would have asked Packagist
for a development version of a package that is not published there.

The root composer.json now replaces it like its siblings, and the lock
file carries the content hash that follows from it.

Still to do outside this repository, per Documentation/Developer/
MonoRepository.rst: create TYPO3-Documentation/t3docs-typo3-docs-theme-md,
give the php-based-rendering-bot team maintain rights, and register it on
Packagist. The split has to run once on a push before a tag is made.

Signed-off-by: lina.wolf
Assisted-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant